To Err Is Human: Correlating Fmri Decoding and Behavioral Errors to Probe the Neural Representation of Natural Scene Categories

New multivariate methods for the analysis of functional magnetic resonance imaging (fMRI) data have enabled us to decode neural representations of visual information with unprecedented fidelity. But how do we know if humans make use of the information that we decode from the fMRI data for their behavioral response? In this chapter we propose a method for correlating the errors from fMRI decoding with the errors made by subjects in a behavioral task. High correlations suggest that subjects use information that is closely related to the content of the fMRI signal to make their behavioral response. We demonstrate the viability of this method using the example of natural scene categorization. Humans are extremely efficient at categorizing natural scenes (such as forests, highways, or beaches), despite the fact that different classes of natural scenes often share similar image statistics. Where and how does this happen in the brain? By applying multi-voxel pattern analysis to fMRI data recorded while subjects viewed natural scenes we found that the primary visual cortex (V1), the parahippocampal place area (PPA), retrosplenial cortex (RSC), and the lateral occipital complex (LOC) all contain information that distinguishes among natural scene categories. Correlation of the decoding errors with errors made by the subjects in a behavioral experiment revealed that only the information about scene categories contained in the PPA, RSC, and LOC is directly related to behavior, but not the information in area V1. A match between behavioral performance and accuracy of decoding scene categories from the PPA and RSC for two manipulations of the stimuli (scene inversion and quality of category exemplar) underscores the central role of these two areas in natural scene categorization.